Add support for selected entities and redirects in add-data workflow by gibahjoe · Pull Request #132 · digital-land/async-request-backend

gibahjoe · 2026-07-20T14:57:17Z

What type of PR is this? (check all applicable)

Description

Adds selected entity handling to the add-data pipeline. Selected entities are prioritised for entity number assignment, new-entities reports only selected rows, and all-entities reports the full set of newly assigned rows.

Also adds old-entity redirect generation from selected redirects and duplicate candidates. Complete duplicate matches and single matches above 85% are automatically redirected, with duplicate evidence included in old-entity.notes.

Related Tickets & Documents

Ticket Link
Related Issue #
Closes #

QA Instructions, Screenshots, Recordings

Run the focused pipeline tests:

cd request-processor
./venv/bin/python -m pytest tests/unit/src/application/core/test_pipeline.py::test_fetch_add_data_response_includes_selected_old_entity_redirects tests/unit/src/application/core/test_pipeline.py::test_create_old_entity_redirects_from_selected_redirects tests/unit/src/application/core/test_pipeline.py::test_create_auto_old_entity_redirects_for_complete_matches tests/unit/src/application/core/test_pipeline.py::test_create_auto_old_entity_redirects_for_single_matches_above_85_percent tests/unit/src/application/core/test_pipeline.py::test_create_auto_old_entity_redirects_ignores_existing_redirects_and_unselected_entities -q

Run formatting check:

cd request-processor
./venv/bin/python -m black --check src/application/core/pipeline.py tests/unit/src/application/core/test_pipeline.py

Added/updated tests?

Yes
No, and this is why:
I need help with writing tests

[optional] Are there any post deployment tasks we need to perform?

None.

[optional] Are there any dependencies on other PRs or Work?

None known.

Summary by CodeRabbit

New Features
- Added add-data inputs to exclude specific references from entity selection and redirect generation.
- Added support for explicitly selecting redirects, alongside automatic redirects for likely duplicates.
- Enhanced add-data responses to include selected new entities and the generated old-entity redirect rows.
Bug Fixes
- Improved behaviour when selection inputs are missing, empty, null, or include invalid/unassigned redirect data.
- Ensured redirect results are de-duplicated and excluded selections are respected.
Tests
- Added unit tests covering selection filtering, assignment prioritisation, and redirect generation scenarios.

- Introduced `selected_entities` and `selected_redirects` parameters in relevant functions and methods across the pipeline, workflow, and tasks modules. - Enhanced entity assignment logic to prioritize selected entities. - Updated tests to validate the new functionality and ensure correct behavior with selected entities and redirects.

coderabbitai · 2026-07-20T14:57:45Z

Warning

Review limit reached

@gibahjoe, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 49 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 10ebb477-a375-409a-ba67-00c94d903541

📥 Commits

Reviewing files that changed from the base of the PR and between 6f8b97c and 7432667.

📒 Files selected for processing (1)

request-processor/src/application/core/pipeline.py

Walkthrough

Changes

Add-data selection and redirects

Layer / File(s)	Summary
Selection request contract and wiring `request_model/schemas.py`, `request-processor/src/application/core/workflow.py`, `request-processor/src/tasks.py`, `request-processor/tests/unit/src/application/core/test_workflow.py`, `request-processor/tests/unit/src/test_tasks.py`	Add-data requests accept optional excluded references and selected redirects, which are forwarded through the task and workflow layers.
Selected entity assignment and transformation `request-processor/src/application/core/pipeline.py`, `request-processor/tests/unit/src/application/core/test_pipeline.py`	Excluded references are assigned after non-excluded entries and used to filter entity-organisation and new-entity summaries.
Explicit and automatic redirect generation `request-processor/src/application/core/pipeline.py`, `request-processor/tests/unit/src/application/core/test_pipeline.py`	Redirect helpers support explicit selections, duplicate matching, similarity thresholds, evidence notes, validation, dates, and de-duplication.
Selection-aware pipeline response `request-processor/src/application/core/pipeline.py`, `request-processor/tests/unit/src/application/core/test_pipeline.py`	The response includes filtered entity breakdowns and merged `old-entity` redirect rows.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant add_data_task
  participant add_data_workflow
  participant fetch_add_data_response
  participant RedirectHelpers
  Client->>add_data_task: Submit excluded_references and selected_redirects
  add_data_task->>add_data_workflow: Forward selection parameters
  add_data_workflow->>fetch_add_data_response: Request selection-aware response
  fetch_add_data_response->>RedirectHelpers: Build and merge old-entity redirects
  RedirectHelpers-->>fetch_add_data_response: Return redirect rows
  fetch_add_data_response-->>add_data_workflow: Return pipeline summary
  add_data_workflow-->>add_data_task: Return add-data response

Possibly related PRs

digital-land/async-request-backend#123: Adds duplicate candidate detection used by this PR’s automatic old-entity redirect generation.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: adding support for selected entities and redirects in the add-data workflow.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch 2661-complex-batch-assign---part-3-choose-which-rows-to-assign-entities-to

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

…ich-rows-to-assign-entities-to

coderabbitai

🧹 Nitpick comments (1)

request-processor/src/application/core/pipeline.py (1)
82-85: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Remove unused selected_entity_ids.

selected_entity_ids is computed here but never referenced afterwards in _create_old_entity_redirects - the exclusion is already enforced via entity_by_reference, which is built from the same selected_new_entities. Flake8 flags this as F841.
🧹 Proposed fix
     selected_new_entities = _filter_selected_entities(new_entities, excluded_references)
-    selected_entity_ids = {
-        str(entity.get("entity", "")).strip() for entity in selected_new_entities
-    }
 
     old_entity_rows = []
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@request-processor/src/application/core/pipeline.py` around lines 82 - 85,
Remove the unused selected_entity_ids computation from
_create_old_entity_redirects, leaving selected_new_entities and the existing
entity_by_reference-based exclusion logic unchanged.
Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@request-processor/src/application/core/pipeline.py`:
- Around line 82-85: Remove the unused selected_entity_ids computation from
_create_old_entity_redirects, leaving selected_new_entities and the existing
entity_by_reference-based exclusion logic unchanged.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 905c7db7-7f44-4b74-9a4a-1b9aba82edcb

📥 Commits

Reviewing files that changed from the base of the PR and between debe04e and 9d2bbaa.

📒 Files selected for processing (7)

request-processor/src/application/core/pipeline.py
request-processor/src/application/core/workflow.py
request-processor/src/tasks.py
request-processor/tests/unit/src/application/core/test_pipeline.py
request-processor/tests/unit/src/application/core/test_workflow.py
request-processor/tests/unit/src/test_tasks.py
request_model/schemas.py

🚧 Files skipped from review as they are similar to previous changes (1)

request-processor/tests/unit/src/test_tasks.py

…ty retrieval in _create_old_entity_redirects

pooleycodes

LGTM

gibahjoe linked an issue Jul 20, 2026 that may be closed by this pull request

Complex Batch Assign - Part 3: Choose which rows to assign entities to digital-land/config#2661

Closed

gibahjoe added 2 commits July 20, 2026 16:05

Merge branch 'main' into 2661-complex-batch-assign---part-3-choose-wh…

80b0150

…ich-rows-to-assign-entities-to

Add docstrings to several functions in pipeline.py for improved clarity

a96405c

gibahjoe temporarily deployed to development July 21, 2026 08:03 — with GitHub Actions Inactive

Refactor entity selection logic

9d2bbaa

coderabbitai Bot reviewed Jul 21, 2026

View reviewed changes

gibahjoe added 3 commits July 22, 2026 08:54

Refactor formatting of excluded_references assignment in add_data_task

6f8b97c

Remove redundant redirect reference functions and streamline old enti…

3821c79

…ty retrieval in _create_old_entity_redirects

Remove redundant selection of entity IDs in _create_old_entity_redirects

7432667

gibahjoe temporarily deployed to development July 22, 2026 08:10 — with GitHub Actions Inactive

pooleycodes approved these changes Jul 22, 2026

View reviewed changes

gibahjoe merged commit 2741c6f into main Jul 23, 2026
8 checks passed

gibahjoe deleted the 2661-complex-batch-assign---part-3-choose-which-rows-to-assign-entities-to branch July 23, 2026 07:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add support for selected entities and redirects in add-data workflow - #132

Add support for selected entities and redirects in add-data workflow#132
gibahjoe merged 7 commits into
mainfrom
2661-complex-batch-assign---part-3-choose-which-rows-to-assign-entities-to

gibahjoe commented Jul 20, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jul 20, 2026 •

edited

Loading

Review limit reached

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Uh oh!

pooleycodes left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

gibahjoe commented Jul 20, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this? (check all applicable)

Description

Related Tickets & Documents

QA Instructions, Screenshots, Recordings

Added/updated tests?

[optional] Are there any post deployment tasks we need to perform?

[optional] Are there any dependencies on other PRs or Work?

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

pooleycodes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gibahjoe commented Jul 20, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 20, 2026 •

edited

Loading